Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.

This PR is useful for any tool or developer that uses linux perf to analyse .NET applications. Specifically, support is being added for use by Arm Performix, with the aim of using this to help migrate users.

Example output of perf annotate showing the line numbers from an example C# that was run through perf:

This PR was entirely written by codex. I've spent some time testing and reviewing the code. It looks sensible and works for me, however I don't claim to be familiar with some of these areas of CoreCLR.

@JulieLeeMSFT @dhartglassMSFT

Still need to add tests. But, taking this out of draft to get some comments to see if this is in the right direction

@noahfalk, @brianrob, @steveisok, PTAL if it aligns with our diagnostics story. This change is related to the Performix tool supporting source annotation.

I'm hesitant that trying to do this source resolution at runtime will necessarily give the overall scenario characteristics you are looking for but we should discuss.

1. Do you expect all PDBs for relevant modules to be present on the disk when the application is running? Deploying PDBs is not typical for .NET apps and most other .NET profiling scenarios I can think of don't rely on this. You would need every developer who wants to use the profiler to do some manual deployment steps and/or create custom builds.
2. How much interference from the profiling code are you willing to tolerate? Searching for PDBs, loading them into memory, and parsing them is going to execute down different non-trivial codepaths and allocate a bunch of memory.
3. For self-contained apps I don't recall if anything is going to ensure the PDB parsing logic is always linked in. We'd need to investigate that or accept that on some apps it just wouldn't work.

The alternative that I've seen other profilers do is PDB/source resolution as a post-processing step after the trace is collected rather than at runtime. It requires at runtime capturing IP->IL offset mappings, then later using PDBs to convert IL -> source line info. Its probably a bit more complex to create initially but gives a higher fidelity result and supports acquiring symbols out-of-band such as from a symbol server. A couple possibilities on how that could be built:

• if there is a way to smuggle IL information into a jitdump file it could be post-processed. I'm not aware that jitdump supports storing that kind of data so probably what this really means is creating a custom file format initially that turns into a valid jitdump after post-processing.
• instead of trying to leverage the runtime's jitdump file writing capability we could use completely different mechanisms to write the initial file such as an EventPipe trace or a custom profiling plugin.

I'm hesitant that trying to do this source resolution at runtime will necessarily give the overall scenario characteristics you are looking for but we should discuss.

Thanks.

1. Do you expect all PDBs for relevant modules to be present on the disk when the application is running? Deploying PDBs is not typical for .NET apps and most other .NET profiling scenarios I can think of don't rely on this. You would need every developer who wants to use the profiler to do some manual deployment steps and/or create custom builds.

Even if we did this as a post processing step instead (or in any other conceivable way) then the source code would need to be present at the point of processing.

This is all hidden behind an additional config variable (DOTNET_PerfMapEmitDebugInfo) which is off by default, so shouldn't cause any overhead for anyone who doesn't want this.

2. How much interference from the profiling code are you willing to tolerate? Searching for PDBs, loading them into memory, and parsing them is going to execute down different non-trivial codepaths and allocate a bunch of memory.

Agreed, this would be a concern as we want this to effect performance as little as possible.

This will overhead will happen at jit compilation time, not execution time. So mostly startup would be effected.

Looking at my test app, running it under Linux perf, with DOTNET_PerfMapEnabled=2 it took ~5.3 seconds. With DOTNET_PerfMapEmitDebugInfo=1 enabled too, it jumps to ~5.8 seconds (almost 10%). Increasing the internal loops in the test app took it to 44.8 secs and 45.3 seconds (now 1%).

Do you have an suggestions as to other ways to measure the impact?

Or at what level you'd consider this to be too much?

3. For self-contained apps I don't recall if anything is going to ensure the PDB parsing logic is always linked in. We'd need to investigate that or accept that on some apps it just wouldn't work.

Ok.

The alternative that I've seen other profilers do is PDB/source resolution as a post-processing step after the trace is collected rather than at runtime. It requires at runtime capturing IP->IL offset mappings, then later using PDBs to convert IL -> source line info. Its probably a bit more complex to create initially but gives a higher fidelity result and supports acquiring symbols out-of-band such as from a symbol server. A couple possibilities on how that could be built:

• if there is a way to smuggle IL information into a jitdump file it could be post-processed. I'm not aware that jitdump supports storing that kind of data so probably what this really means is creating a custom file format initially that turns into a valid jitdump after post-processing.

Post processing would be a valid way of doing this

• instead of trying to leverage the runtime's jitdump file writing capability we could use completely different mechanisms to write the initial file such as an EventPipe trace or a custom profiling plugin.

That would have the same issue that it'd be during runtime and so would effect performance?

The alternative that I've seen other profilers do is PDB/source resolution as a post-processing step after the trace is collected rather than at runtime. It requires at runtime capturing IP->IL offset mappings, then later using PDBs to convert IL -> source line info. Its probably a bit more complex to create initially but gives a higher fidelity result and supports acquiring symbols out-of-band such as from a symbol server. A couple possibilities on how that could be built:

Downside to this is you still have overhead. You remove all the PDB reading, but it still needs to write a similar amount of information out to the dump.

In addition, using the PR as is, this will work out of the box for any Linux app which uses perf. Using IL in jitdump or events is a bespoke mechanism. We'd essentially have to write a tool/library that does all the processing, then that needs distributing and fitting into any app that wants this information.